Method and apparatus for classifying video data

ABSTRACT

A method of classifying video data representing activity within a space to be monitored. A method comprises storing video data obtained from a camera configured to monitor the space. Sensor data indicative of a condition occurring within the space is obtained, and a plurality of program elements are defined within the video data. Each program element has an associated classification code, and each classification code is selected using the sensor data.

The present invention relates to a method and apparatus for classifyingvideo data. More particularly, the invention relates to the use ofsensor data to define classification codes which are applied toprogramme elements within the video data.

It is well known to use a video camera to capture video data, and to usethat video data to monitor an area for security purposes. For example,in a simple system a video camera may constantly capture activity withina particular area, such that if it is determined that unauthorisedactivity has taken place, a security operative can review video data toobtain further information as to the nature of the unauthorisedactivity.

In more complex systems, a plurality of video cameras may be used tocapture video data from different areas. In such a system, movementsensors may be used to select which of the plurality of video cameras isused to capture video data at a particular time, so as to generate asingle stored stream of video data. Such systems are clearly moreconvenient where a large number of disparate areas are to be monitored.It is also known in the prior art to use motion and other sensors todetermine the way in which a particular video camera should beactivated. For example zoom and angle adopted by a particular videocamera may be determined using such sensor data.

Although the known security systems described above provide a mechanismby which video data can be captured and reviewed to determine the natureof unauthorised activity, it will be appreciated that such systems aredisadvantageous given that relatively large quantities of video datamust be reviewed so as to accurately locate the activity of interestwithin the video data. If, as is usual, the video data is captured ontraditional video cassettes, typically fast forward and rewindoperations will be performed in a relatively random manner in an attemptto identify the correct position in the video data.

It is an object of the present invention to provide a method ofclassifying video data which obviates or mitigates at least some of theproblems outlined above.

To assist in understanding of the invention, the term “programmeelement”, as that term is used in this document, is used to mean a videoclip. Programme elements can be classified on the basis of any criteriaof interest, such as a level of activity within the video clip.

According to the present invention, there is provided, a method ofclassifying video data representing activity within a space to bemonitored. The method comprises receiving video data from a cameraconfigured to monitor the space. Sensor data indicative of a conditionoccurring within the space is also received, and a plurality ofprogramme elements within said video data are defined. Each programmeelement has an associated classification code, and each classificationcode is selected using the sensor data. Video data representing theprogramme elements is stored, and the associated classification codesare also stored.

By applying classification codes based upon obtained sensor data, aclassified stream of video data is created which can be easily navigatedby a user so as to identify particular parts of the video data on thebasis of the sensor data. Thus, the invention generates video data whichcan be more easily navigated.

The sensor data may be obtained using a motion sensor, a sound sensor,or a pressure sensor. Indeed, the sensor may take any suitable form. Forexample, in some embodiments of the invention the sensor may compriseimage processing means configured to process an image and to outputsensor data indicative of properties of the processed image. Forexample, brightness of captured images may be analysed to generateappropriate sensor data, or images may be analysed to detect movement ofobjects or people within the space which is being monitored.

The method may further comprise obtaining video data from a plurality ofcameras, and storing data from one of said plurality of cameras selectedon the basis of the sensor data. Here, a single stream of video data maybe generated and at each point in time a camera recording video dataindicating activity may provide the video data. The classification codeassociated with each programme element may identify the camera of saidplurality of cameras which was used to obtain the video data making upeach programme element.

The method may comprise obtaining sensor data, and storing video data ifbut only if said sensor data satisfies a predetermined condition.

The classification codes may be selected from a predetermined set ofclassification codes. The classification codes may represent values on ascale extending from a high value to a low value. The scale mayrepresent relative activity within the space being monitored.

The method may comprise deleting video data representing some programmeelements, the video data to be deleted being selected on the basis ofsaid classification codes. The video data to be deleted may be videodata representing periods of relatively low activity within the spacebeing monitored.

The method may further comprise receiving user selection of aclassification code, and displaying programme elements associated withthat classification code. User input representing a request to displayprogramme elements representing periods of relatively high activitywithin the space being monitored may be received, at least oneclassification code associated with programme elements representingperiods of relatively high activity within the space being monitored maybe determined, and programme elements associated with the determinedclassification codes may then be displayed. The determining may comprisecalculating an average level of activity for all programme elements, anddetermining classification codes representing periods of above averageactivity.

According to further aspects of the present invention, there is provideda data carrier carrying computer programme code means to cause acomputer to carry out the method set out above.

The invention also provides an apparatus for classifying video datarepresenting activity within a space to be monitored. The apparatuscomprises a programme memory storing processor readable instructions,and a processor configured to read and execute instructions stored insaid programme memory. The processor readable instructions compriseinstructions controlling the processor to carry out the method describedabove.

According to a further aspect of the present invention, there isprovided an apparatus for classifying video data representing activitywithin a space to be monitored. The apparatus comprises a storage deviceconfigured to store video data obtained from a video camera configuredto monitor the space, input means configured to receive sensor dataindicative of a condition occurring within the space, and means fordefining a plurality of programme elements within said video data eachprogramme element having an associated classification code. Theclassification codes are selected using said sensor data.

The apparatus may further comprise a video camera, and may furthercomprise at least one sensor. The sensor may be a motion sensor, apressure sensor or a sound sensor.

Embodiments of the present invention will now be described, by way ofexample, with reference to the accompanying drawings, in which:

FIG. 1 is a schematic illustration of a first embodiment of the presentinvention;

FIG. 2 is a flow chart showing an overview of operation of the system ofFIG. 1;

FIG. 3 is flow chart showing in more detail a method of determiningclassification codes in the method of FIG. 2; and

FIG. 4 is a flow chart showing an alternative method of determiningclassification codes in the flow chart of FIG. 2;

FIG. 5 is a schematic illustration of a second embodiment of the presentinvention; and

FIG. 6 is flow chart showing an overview of operation of the system ofFIG. 5.

Referring to FIG. 1, a controller 1 receives video data from a videocamera 2, and sensor data from a sensor 3. In a first embodiment of thepresent invention, the sensor 3 is a motion sensor such as a passiveinfrared detector (PIR). The controller 1 is also connected to anon-volatile storage device 4 which can suitably take the form of a harddisk drive or other similar device. The controller 1 writes video data 5and classification data 6 to the storage device 4. The classificationdata 6 comprises classification codes associated with programme elementscomprised in the video data 5.

Referring to FIG. 2, there is illustrated a flow chart providing anoverview of operation of the controller 1. At step S1 the controller 1simply passes video data received from the video camera 2 to the storagedevice 4, for storage as video data 5. At step S2, the controller 1 usessensor data obtained from the sensor 3 to generate classification codeswhich can be applied to the video data 5. These classification codes arestored on the storage device 4 as classification data 6 (step S3).

The classification data 6 is used to identify programme elements withinthe video data 5 and classifies programme elements in terms of theirlevel of activity within an area of being monitored by the video camera2. Thus, an operator viewing the video data 5 can use the classificationdata 6 to identify periods of relatively high activity. This hasparticular value in a security system in which a security operative canquickly locate parts of the video data which perhaps relate tosuspicious activity.

Referring to FIG. 3, a method of determining classification codescarried out by the controller 1 at step S2 of FIG. 2 is illustrated. Atstep S4 a time parameter is initialised to t, where t is a time markerat a current position within the video data 5. At step S5 a countervariable i is initialised to zero. Steps S6, S7 and S8 are then repeatedN times to obtain sensor data from the sensor 3. At each iteration ofthe loop, data is sampled and stored from the sensor 3 (step S6). Atstep S7, the counter variable i is incremented. At step S8 a check ismade to determine whether the counter variable i is still less than N.If the condition of step S8 is satisfied, a further iteration of theloop is performed, and accordingly processing returns to step S6. If thecondition of step S8 is not satisfied, it can be determined that Nsample values have been processed and processing then passes to step S9.

In the described embodiment of the present invention, the sensor 3outputs a binary value. That is, the sensor 3 outputs a ‘1’ if activityis sensed, and ‘0’ if no activity is sensed. The binary values obtainedand stored at step S6 are summed at step S9. At step S10, the result ofthe summation of step S9 is used to determine a classification code fora programme element beginning at time t. The summation will take a valuein the range 0 to N. Thus, if N+1 different classification codes aredefined, the result of the summation can itself be the classificationcode. However, if a more limited scale is desired it will be readilyapparent how the value in the range 0 to N+1 can be used to determine aclassification code at step S10. The classification code determined atstep S10 is stored as part of classification data 6 at step S11. Theclassification code can suitably be stored as part of a tuple of theform:(C,N,t,t _(s))Where:

-   -   C is a classification code determined as described above;    -   N is a number of samples of sensor data obtained from the sensor        3 used to compute that classification code;    -   t is a time point within the video data 5 at which programme        element classified using the same classification code begins;        and    -   t_(s) is a sample time (i.e. a time interval between the capture        of subsequent samples of the N sample).        Using data within the tuple set out above it will be appreciated        that a unique portion of the video data 5 is identified with        which the classification code C is associated. That portion        begins at time t and ends at time (t+N×t_(s)).

At step S12 the time parameter is updated according to an equation:Time=(t+N×t _(s))+t _(s)Where the parameters take the values set out above.

Processing then returns to step S5 where i is set to 0, and aclassification code for a programme element beginning at time[t+(N×t_(s))+t_(s)] is determined in the manner described above.

Using the process described with reference to FIG. 3, it will beappreciated that a plurality of programme elements of equal duration aredefined, each having an associated classification code. When reviewingvideo data classified using the process of FIG. 3 an operator can easilylocate periods of relatively high activity by specifying classificationcodes of relatively high value to cause the controller 1 to retrieveprogramme elements classified using that classification code andtherefore representing periods of relatively high activity.

Referring to FIG. 4, there is illustrated an alternative process to thatillustrated in FIG. 3. At step S13 a time parameter is again set to t.At step S14 sensor data from the sensor 3 is captured, and at step 15the captured value is examined to determine whether the captured valueis indicative of activity. That is, if the sensor 3 outputs binary datain the manner described above a check is made to determine whether thecaptured value is equal to 1. If the captured value is equal to 1 thevalue of the time parameter is stored in a list at step S16 the timeparameter is incremented at step S17 and the process of S14 to S16 isagain repeated. If activity is not detected at step S15, processingpasses directly to step S17.

Using the process described with reference to FIG. 4, the classificationdata 6 comprises a list of time values at which the sensor 3 output is abinary ‘1’ value. An operator viewing video data classified in thismanner can then use a summation process of the type described withreference to FIG. 3 so as to determine periods of relatively highactivity. It will be appreciated that storing raw sensor values asopposed to generalised classification codes allows an operator greaterflexibility. Indeed, an operator may wish to view all programme elementsdefined by more than two sensor activations within a predetermined timeperiod.

In the embodiments of the invention described above, the sensor 3 hasbeen described as a motion sensor, and more particularly PIR detector.It will be appreciated that other motion detectors such as, for example,active infrared sensors are equally applicable for use in the invention.Furthermore, the sensor 3 need not be a motion sensor. In someembodiments of the present invention the sensor 3 takes the form of amicrophone, and a sound level detected by the microphone is used asinput to the controller 1. In such embodiments of the invention, theoutput of the sensor 3 may not be a binary output, but instead be ananalogue value indicative of sound value measured in decibels. In suchembodiments of the invention it will be readily apparent to one skilledin the art that a simple thresholding algorithm can be used so as todefine binary values of the type described above. That is, if the soundlevel is above a predetermined value a ‘1’ value is generated, while ifthe sensor output is below the predetermined value a ‘0’ value isgenerated. The present invention is also applicable where the sensor 3is a pressure sensor placed, for example, on a floor and thus detectingmovement within a room.

It will also be appreciated that the sensor 3 can be a plurality ofdifferent sensors, the outputs of which are combined by the controller 1to generate classification codes. For example a motion detector can beused to compute classification code in the manner described withreference to FIG. 3. A sound level of a microphone can then beadditionally analysed as described above using a relatively highthreshold. If it is the case that the detected sound level is, at anytime, in excess of the high threshold a special classification codeindicating suspicious activity may be applied to the video data.

The present invention is also applicable to a system in which recordingby the video camera 2 is triggered by the sensor 3. Classification codescan then be applied to captured video data in the manner describedabove. The present invention is also applicable to systems in aplurality of video cameras monitoring different spaces, in which it isdesired to record a single stream of video data. In such systems, thesingle stream of video data is, at any one time, generated by the outputof a single video camera. This video camera from which video data isstored can suitably be selected by appropriately positioning varioussensors and using this data as the basis for camera selection. In suchembodiments of the invention, the classification data relating to thevideo data 5 can simply indicate the camera from which the appropriatevideo data was captured. This can be determined on the basis of whichsensor triggered recording. In such embodiments of the presentinvention, an operator may know that suspicious activity occurred withinan area monitored by a particular camera, and can use the classificationdata 6 to locate parts of the video data 5 which were provided by thatvideo camera.

Referring now to FIG. 5, there is illustrated a modified version of thesystem of FIG. 1. In the system illustrated in FIG. 5, the controller 1comprises a video data buffer 7 and a classification code determiningmeans 8. It can be seen that video data from the video camera 2 ispassed directly to the video data buffer 7, while sensor data from thesensor 3 is passed directly to the classification code determining means8. The classification data 6 is provided by the classification codedetermining means 8, and the video data 5 is provided from the videodata buffer 7.

FIG. 6 illustrates an overview of operation of the system of FIG. 5. Atstep S18 video data is received by the buffer 7 from the video camera 2.At step S19, classification code determining means 8 determines aclassification code for a portion of the video data stored in the videodata buffer 7. This can conveniently be carried out using a process asillustrated in FIG. 3, and described above. At step S20, theclassification code determined at step S19 is compared with apredetermined criteria. This criteria may be, for example, apredetermined level of activity. If the condition of step S20 issatisfied, processing passes to step S21 where the video data associatedwith the classification code generated at step S19 is written from thevideo data buffer 7 to the video data 5 stored on the non-volatilestorage device 4. Similarly, at step S22 the classification codedetermined by the classification code determining means 8 is written tothe classification data 6. Having copied video data from the video databuffer 7 to the video data 5, the video data buffer 7 is cleared at stepS23. If the predetermined criteria of step S20 is not satisfied thevideo data is simply deleted from the video data buffer 7 (step S23),and is not written to the non-volatile storage device 4.

The video data buffer 7 can conveniently be implemented as a pluralityof buffers arranged in parallel, each holding a different programmeelement. In this way, a plurality of programme elements may be stored inthe video data buffer 7 before a decision is made as to whether suchprogramme elements should be stored as part of the video data 5 storedon the non-volatile storage device 4. Each of the plurality of buffersis provided with a flag indicating whether its data can be overwritten.This flag is set when data is written to a respective buffer, and unsetwhen data is copied to the video data 5 or a decision is taken that theprogramme element is not to be stored. The implementation of such abuffering arrangement will be readily apparent to one of ordinary skillin the art.

It can be seen that the embodiment of the invention as described withreference to FIGS. 5 and 6 provides a convenient mechanism for storingonly portions of video data which represent periods of relatively highactivity within the space to be monitored, thereby saving storage space.Video data that is stored is classified in the manner described above.

In some embodiments of the present invention, the classification data 6may periodically be used to identify portions of the video data 5representing periods of relatively low activity. Having identified suchperiods of the video data 5, this video data may be deleted so as tocreate more free space on the non-volatile storage device 4.

The manipulation and viewing of a classified stream of video data can becarried out using various developments made by the present applicant anddescribed in US Patent Application publication No. US2004/0070594(application Ser. No. 10/435,178) the contents of which are hereinincorporated by reference. The adaptation of such techniques to theviewing of video data captured using the methods described above will bereadily apparent.

In particular, in some embodiments of the present invention, a user maysimply specify that they are interested in periods of the recorded videodata 5 which represent periods of relatively high activity. On receivingsuch a request, the controller 1 may interrogate the classification data6 to determine an average activity level using stored classificationcodes. Having determined such an average activity level classificationcodes representing periods of above average activity are identified, andthe corresponding video data is received from the video data 5.

The embodiments of the invention described above are intended only to beexamples of ways in which the present invention may be put into effect,and are in no way intended to limit the scope of the claims. Indeed, itwill be readily apparent to one of ordinary skill in the art thatvarious amendments can be made to the embodiments described abovewithout departing from the spirit and scope of the present invention asdefined by the appended claims.

The invention claimed is:
 1. A method of classifying video datarepresenting activity within a space to be monitored, the methodcomprising: receiving video data from a camera configured to monitor thespace; receiving sensor data indicative of a condition occurring withinthe space; defining a plurality of programme elements within said videodata, each programme element having an associated classification codeselected using a plurality of samples of said sensor data obtained by asensor during a time period of capture of the programme element by thecamera; storing video data representing said programme elements; andstoring said classification codes associated with said programmeelements; wherein said classification codes represent values on a scaleextending from a low value to a high value, said scale representingrelative activity within the space being monitored.
 2. A methodaccording to claim 1, wherein said sensor data is obtained using amotion sensor.
 3. A method according to claim 1, wherein said sensordata is obtained using a sound sensor.
 4. A method according to claim 1,wherein said sensor data is obtained using a pressure sensor.
 5. Amethod according to claim 1, further comprising: receiving video datafrom a plurality of cameras; and storing data from one of said pluralityof cameras selected on the basis of said sensor data.
 6. A methodaccording to claim 5, wherein the classification code associated witheach programme element identifies the camera of said plurality ofcameras which was used to obtain the video data making up said programmeelement.
 7. A method according to claim 1, wherein storing video datarepresenting said programme elements comprises: determining whether eachclassification code satisfies a predetermined condition; and storing aprogramme element if but only if its associated classification codesatisfies said predetermined condition.
 8. A method according to claim1, wherein said classification codes are selected from a predeterminedset of classification codes.
 9. A method according to claim 1, whereineach classification code is selected in dependence upon a number ofactivations of the sensor occurring within a corresponding predeterminedtime period.
 10. A method according to claim 1, further comprising:deleting video data representing some programme elements, wherein videodata to be deleted is selected on the basis of said classificationcodes.
 11. A method according to claim 1, wherein video data to bedeleted is video data representing periods of relatively low activitywithin the space being monitored.
 12. A method according to claim 1,further comprising: receiving user selection of a classification code;and displaying programme elements associated with said classificationcode.
 13. A method according to claim 9, further comprising: receivinguser input representing a request to display programme elementsrepresenting periods of relatively high activity within the space beingmonitored; determining at least one classification code associated withprogramme elements representing periods of relatively high activitywithin the space being monitored; and displaying programme elementsassociated with the determined classification codes.
 14. A methodaccording to claim 13, wherein said determining comprises: calculatingan average level of activity for all programme elements; and determiningclassification codes representing periods of above average activity. 15.A non-transitory computer-readable memory storing computer program codeto cause a computer to carry out the method of claim
 1. 16. An apparatusfor classifying video data representing activity within a space to bemonitored, the apparatus comprising: a program memory storing processorreadable instructions; and a processor configured to read and executeinstructions stored in said program memory; wherein said processorreadable instructions comprise instructions controlling the processor tocarry out the method of claim
 1. 17. An apparatus for classifying videodata representing activity within a space to be monitored, the apparatuscomprising: a storage device configured to store video data obtainedfrom a video camera configured to monitor the space; a controllerconfigured to receive sensor data indicative of a condition occurringwithin the space; wherein the controller is configured to define aplurality of programme elements within said video data, each programmeelement having an associated classification code selected using aplurality of samples of said sensor data obtained by a sensor during atime period of capture of the programme element by the video camera;wherein said classification codes represent values on a scale extendingfrom a low value to a high value, said scale representing relativeactivity within the space being monitored.
 18. An apparatus according toclaim 17, further comprising the video camera.
 19. An apparatusaccording to claim 17, further comprising the sensor.
 20. An apparatusaccording to claim 19, wherein said sensor is a motion sensor.
 21. Anapparatus according to claim 19, wherein said sensor is a pressuresensor.
 22. An apparatus according to claim 19, wherein said sensor is asound sensor.
 23. A method of classifying video data representingactivity within a space to be monitored, the method comprising:receiving video data from a camera configured to monitor the space;receiving sensor data indicative of a condition occurring within thespace; defining a plurality of programme elements within said videodata, each programme element having an associated classification codeselected using a plurality of samples of said sensor data obtained by asensor during a time period of capture of the programme element by thecamera; storing video data representing said programme elements; andstoring said classification codes associated with said programmeelements.
 24. An apparatus for classifying video data representingactivity within a space to be monitored, the apparatus comprising: astorage device configured to store video data obtained from a videocamera configured to monitor the space; a controller configured toreceive sensor data indicative of a condition occurring within thespace; wherein the controller is configured to define a plurality ofprogramme elements within said video data, each programme element havingan associated classification code selected using a plurality of samplesof said sensor data obtained by a sensor during a time period of captureof the programme element by the video camera.